Camera and specimen alignment to facilitate large area imaging in microscopy

ABSTRACT

A microscope system and method allow for a desired x′-direction scanning along a specimen to be angularly offset from an x-direction of the XY translation stage, and rotates an image sensor associated with the microscope to place the pixel rows of the image sensor substantially parallel to the desired x′-direction. The angle of offset of the x′-direction relative to the x-direction is determined and the XY translation stage is employed to move the specimen relative to the image sensor to different positions along the desired x′-direction without a substantial shift of the image sensor relative to the specimen in a y′-direction, the y′-direction being orthogonal to the x′ direction of the specimen. The movement is based on the angle of offset.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/457,470, filed Feb. 10, 2017.

FIELD OF THE INVENTION

The present invention generally relates to microscopy imagingtechniques. In particular, the present invention relates to apparatusand methods for aligning a camera/image sensor with a specimen so as toprecisely scan across the specimen along a desired direction.

BACKGROUND OF THE INVENTION

In microscopy, an area of interest in a specimen to be imaged is oftenlarger than can be displayed by taking a single image with themicroscope. Thus scanning techniques are employed to image an entiredesired area. In automated scanning, the specimen is moved under theobjective lens of the microscope by an XY translation stage so that themicroscope can scan across the desired area, with multiple images beingcollected and then aggregated or stitched to form a single larger image.This stitching can be accomplished using standard software techniques orby ensuring that images are taken at specific locations with veryprecise stage movement feedback such that, when a first image is taken,the stage moves exactly the distance equal to the width of the firstimage (and without movement in the height direction) and a second imageis taken to them be joined at the common border. If precise enough, theleft edge of first image will then exactly mate and compliment the rightedge of for the second image.

It is often advantageous to align the camera pixels to a specificorientation relative to the specimen and then scan that specimen in aspecific desired direction and in a manner that maintains the desiredorientation. For example, components on silicon wafers (e.g., microelectronic devices or patterned films such as through photolithography)are often oriented in rows (x-direction) or columns (y-direction), andit is helpful to align the camera pixel rows parallel to a component rowor align the camera pixel columns parallel to a component column to thenaccurately scan along a desired row or column while maintaining theparallel relationship there between.

In the current state of the art the camera pixel orientation is oftenmanually aligned to the XY travel of the stage by visible observation,which, in light of the size scales typically involved, does not providea suitable level of accuracy for many imaging needs. The chances ofaccurately aligning the pixel rows with the x-direction of the stage andthe pixel columns with the y-direction of the stage are very low. Afterthis likely inaccurate alignment, the specimen is rotated relative tothe stage in an attempt to align the pixel rows of the image sensor witha desired x′-direction for scanning the specimen and/or to align thepixel columns with a desired y′-direction for scanning the specimen.That is, the specimen is rotated relative to the XY translation stage inorder to position a desired x′ scanning direction of the specimenparallel to the x-direction movement of the stage and/or position adesired y′ scanning direction of the specimen parallel to they-direction movement of the stage (i.e., the x′-direction andx-direction are intended to be the same and the y′-direction andy-direction are intended to be the same). The x-direction of the XYstage and the rows of pixels of the camera having been previouslyvisually aligned (as are, axiomatically, the y-direction of the stageand the columns of pixels), the movement of the stage in the desiredx-direction or desired y-direction maintains the desired alignment butonly to the extent the manually alignment of the image sensor pixels tothe XY travel of the stage is highly accurate and precise.

Returning to the patterned silicon wafer example, a desired x-directionmight be a row of micro-circuits with this row being aligned parallel tothe x-direction movement of the XY translation stage, and, thus parallelto the rows of pixels of the camera. The row of micro-circuits can thusbe scanned simply by moving the XY translation stage in the x-direction,while the parallel relationship between the rows of pixel and the row ofmicrocircuits is maintained, and the image sensor is not shifted in they′-direction, such that accurate recording and stitching is facilitated.

Thus, accurate results depend upon highly accurate alignment of thecamera pixels, the XY travel of the stage, and the desired x′ and/or y′scanning directions of the specimen. This is difficult to achieve givennormal tolerance in machining and errors inherent in mere visualobservation alignment. If even slightly out of alignment, the imagesensor will be shifted in the x′-direction and/or y′-direction to anunacceptable degree as the specimen is moved by the translation stage,thus frustrating the ease by which images can be analyzed and/orstitched together. Additionally, it is often desired that a specimen beanalyzed with a minimal amount of handling of the specimen. Thus, thereis a need in the art for new methods for aligning and scanning that donot rely on specimen movement and ensure accurate alignment between theimage sensor and the specimen.

SUMMARY OF THE INVENTION

In a first embodiment, the present invention provides a microscopymethod for imaging a specimen along a desired x′-direction of thespecimen. The specimen is placed on an XY translation stage and movableby the XY translation stage so as to have a portion of the specimenplaced within the field of view of an image sensor. The XY translationstage is movable in an x-direction and a y-direction to move thespecimen relative to the image sensor, the image sensor having amultitude of pixels arranged to define pixel rows and pixel columns, thedesired x′-direction of the specimen being angularly offset from thex-direction of the XY translation stage so as to define a slope andangle of offset relative thereto, the image sensor viewing only adiscrete segment of the specimen at a time. The method comprises thesteps of: rotating the image sensor such that the pixel rows aresubstantially parallel with the desired x′-direction of the specimen;determining the angle of offset of the desired x′-direction as comparedto the x-direction of the XY translation stage; establishing a firstposition for the specimen relative to the image sensor as rotated insaid step of rotating, said first position placing at least a portion ofthe specimen within the field of view of the image sensor; and, aftersaid step of determining and said step of establishing, moving thespecimen with the XY translation stage to a second position along thedesired x′-direction, wherein the second position places at least asecond portion of the specimen within the field of view of the imagesensor, and the second position is not substantially shifted in ay′-direction of the specimen, the y′-direction being orthogonal to thex′-direction of the specimen, wherein said step of moving is based uponthe angle of offset determined in said step of determining.

In a second embodiment, the present invention provides a microscopymethod as in any of the embodiments above, wherein said step ofdetermining an angle of offset includes: measuring the x distance and ydistance between a first focal feature and a second focal featurealigned along and thus defining the desired x′-direction, the x distanceand y distance being measured relative to the x-direction andy-directions of the translation stage.

In a third embodiment, the present invention provides a microscopymethod as in any of the embodiments above, wherein said step ofmeasuring the x distance and y distance includes: placing the firstfocal feature so as to overlap with one or more target pixels of theimage sensor, and thereafter moving the specimen to place the secondfocal feature so as to overlap with the same one or more target pixels,said step of measuring the x distance and y distance being the magnitudeof x and y movement of the translation stage (ΔX, ΔY) necessary toachieve said step of moving the specimen to place the second focalfeature so as to overlap with the same one or more target pixels.

In a fourth embodiment, the present invention provides a microscopymethod as in any of the embodiments above, wherein said target pixelsencompass the center of the image sensor.

In a fifth embodiment, the present invention provides a microscopymethod as in any of the embodiments above, wherein said step of rotatingthe image sensor includes identifying an axis-defining feature on thespecimen running in the x′-direction, and using computer vision to alignthe pixel rows substantially parallel to the detectable direction of thespecimen.

In a sixth embodiment, the present invention provides a microscopymethod as in any of the embodiments above wherein said step of rotatingthe image sensor is performed before said step of measuring the xdistance and y distance.

In a seventh embodiment, the present invention provides a microscopymethod as in any of the embodiments above, wherein said step of rotatingthe image sensor includes taking a mosaic of images suitable forcalculating a reference line between the first focal feature and thesecond focal feature and using computer vision to align the pixel rowsof the image sensor with the reference line.

In an eighth embodiment, the present invention provides a microscopymethod as in any of the embodiments above, wherein said step of taking amosaic of images is carried out while carrying out said step ofmeasuring the x distance and y distance.

In a ninth embodiment, the present invention provides a microscopymethod as in any of the embodiments above, wherein, before said step ofrotating the image sensor, the method includes the step of aligning thepixel rows substantially parallel to the x-direction of the XYtranslation stage.

In a tenth embodiment, the present invention provides a microscopymethod as in any of the embodiments above, wherein said step of rotatingthe image sensor includes: identifying an axis-defining feature on thespecimen, the axis-defining feature having a detectable shape running inthe desired x′-direction; and using computer vision to align the pixelrows substantially parallel to the detectable shape, and said step ofdetermining the angle of offset includes: measuring the degrees ofrotation of the image sensor from its position after said step ofaligning the pixel rows substantially parallel to the x-direction of theXY translation stage to its position after said step of rotating theimage sensor.

In an eleventh embodiment, the present invention provides a microscopymethod as in any of the embodiments above, wherein said XY translationstage provides a specimen chuck to hold the specimen, wherein either thespecimen chuck or a specimen placed thereon includes a reference mark,and said step of aligning the pixel rows substantially parallel with thex-direction of the XY translation stage includes: placing the referencemark at a first position within in the field of view of the image sensorand taking image data to determine a first pixel row number for theposition of the reference mark relative to the pixel rows, moving thespecimen chuck along only the x-direction of the XY translation stage toplace the reference mark at a second position within the field of viewof the image sensor and taking image data to determine a second pixelrow number for the position of the reference mark relative to the pixelrows, and, after said steps of placing and moving, rotating the imagesensor to place the reference mark at a third position having a thirdpixel row number that is between said first pixel row number and saidsecond pixel row number.

In a twelfth embodiment, the present invention provides a microscopymethod as in any of the embodiments above, wherein, after said step ofrotating the image sensor to place the reference mark at a thirdposition, said steps of (i) placing the reference mark at a firstposition, (ii) moving the specimen chuck along only the x-direction, and(iii) rotating the image sensor to place the mark at a third positionare repeated until the pixel rows are substantially parallel with thex-direction of the XY translation stage.

In a thirteenth embodiment, the present invention provides a microscopymethod as in any of the embodiments above, wherein said step ofdetermining is carried out after said step of aligning the pixel rowssubstantially parallel with the x-direction of the XY translation stage.

In a fourteenth embodiment, the present invention provides a microscopymethod as in any of the embodiments above, wherein said step of rotatingthe image sensor includes: identifying an axis-defining feature on thespecimen, the axis-defining feature having a detectable shape running inthe desired x′-direction; and using computer vision to align the pixelrows substantially parallel to the detectable shape, and said step ofdetermining the angle of offset includes: measuring the degrees ofrotation of the image sensor from its position after said step ofaligning the pixel rows substantially parallel to the x-direction of theXY translation stage to its position after said step of rotating theimage sensor.

In a fifteenth embodiment, the present invention provides a microscopymethod as in any of the embodiments above, wherein said step ofmeasuring the degrees of rotation of the image sensor includes obtaininga signal output from an instrument rotating the image sensor.

In a sixteenth embodiment, the present invention provides a microscopesystem comprising: a microscope; an image sensor recording image data,said image sensor including pixel rows and pixel columns; an XYtranslation stage; a specimen on said XY translation stage and viewed bysaid image sensor, wherein the XY translation stage is movable in anx-direction and a y-direction to move the specimen relative to the imagesensor, the image sensor having a multitude of pixels arranged to definepixel rows and pixel columns, wherein the specimen presents featuresalong a x′-direction that is angularly offset from the x-direction ofthe XY translation stage so as to define an angle of offset relativethereto, the specimen further including a first focal feature and asecond focal feature, a processor serving to: rotate the image sensorrelative to the specimen such that the pixel rows are parallel with thex′-direction of specimen, move the XY translation stage; determine theangle of offset of the x′-direction as compared to the x-direction ofthe XY translation stage; and scanning across the specimen in thedesired x′ direction by: establishing a first position for the specimenrelative to the image sensor when the pixel rows are parallel with thex′-direction, the first position placing at least a portion of thespecimen within the field of view of the image sensor; and, moving thespecimen with the XY translation stage to a second position along thedesired x′-direction, wherein the second position places at least asecond portion of the specimen within the field of view of the imagesensor, and the second position is not substantially shifted in ay′-direction of the specimen, the y′-direction being orthogonal to thex′-direction of the specimen, wherein the movement is based upon theangle of offset determined by the processor.

In a seventeenth embodiment, the present invention provides a method foraligning pixel rows of an image sensor with the x-direction of an XYtranslation stage, wherein said XY translation stage provides a specimenchuck to hold the specimen, the specimen chuck being moved in ax-direction and a y-direction by the XY translation stage, the methodcomprising the steps of: providing a reference mark on the specimenchuck or on a specimen placed on the specimen chuck; placing thereference mark at a first position within the field of view of the imagesensor and taking image data to determine a first pixel row number forthe position of the reference mark relative to the pixel rows of theimage sensor, moving the specimen chuck along only the x-direction ofthe XY translation stage to place the reference mark at a secondposition within the field of view of the image sensor and taking imagedata to determine a second pixel row number for the position of thereference mark relative to the pixel rows, and, after said steps ofplacing and moving, rotating the image sensor to place the referencemark at a third position having a third pixel row number that is betweensaid first pixel row number and said second pixel row number; wherein,after said step of rotating the image sensor to place the reference markat a third position, said steps of (i) placing the reference mark at afirst position, (ii) moving the specimen chuck along only thex-direction, and (iii) rotating the image sensor to place the mark at athird position are repeated until the pixel rows are substantiallyparallel with the x-direction of the XY translation stage.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic side view of a microscope system in accordancewith this invention;

FIG. 1B a schematic front view of FIG. 1;

FIG. 2A is a schematic side view of a single axis translation stage;

FIG. 2B is a schematic top plan view of the translation stage of FIG. 2A

FIG. 2C is a schematic top plan view of an XY translation stage formedof two single axis translation stages

FIG. 3A is a schematic top plan view of a camera rotator useful in anembodiment of this invention;

FIG. 3B is a schematic side view of the camera rotator of FIG. 3A;

FIG. 4 is a schematic representation of pixels of an image sensor;

FIG. 5 is a schematic representation of a specimen S on an XYtranslation stage (18 a, 18 b) relative to an image sensor 56, showingthe general conditions for application of an embodiment of the presentinvention for determining the angle of offset of a desired x′ directionfor scanning across the specimen as compared to the x-direction of theXY translation stage;

FIG. 6A is a schematic representation of a first step in determining theangle of offset from the general starting conditions of FIG. 5;

FIG. 6B is a schematic representation of a second step for determiningthe angle of offset;

FIG. 7A is a schematic representation of a third step for determiningthe angle of offset, concluding with the ability to determine the slope(ΔX and ΔY) of the x′-direction, and thus its angle of offset;

FIG. 7B is a schematic representation of a rotation step wherein therows of pixels of the image sensor are made to be substantially parallelto the x′-direction;

FIG. 8 is a schematic representation of a method for moving a specimenrelative the image sensor along a desired x′-direction for a desired d′distance;

FIG. 9A is a schematic representation of a reference mark 64 on aspecimen chuck 30 on an XY translation stage (18 a, 18 b), with an imagesensor 56, showing the general conditions for application of anembodiment of the present invention for aligning the pixel rows of animage sensor with the x-direction of the XY translation stage;

FIG. 9B is a schematic representation of a first step in aligning thepixel rows from the general starting conditions of FIG. 5;

FIG. 9C is a schematic representation of a second step in aligning thepixel rows from the general starting conditions of FIG. 5;

FIGS. 10A and 10B together provide a schematic representation of amethod for iteratively rotating an image sensor to align its pixel rowswith the x-direction of the XY translation stage;

FIGS. 11A, 11B, and 11C, together provide a schematic representation ofa method for rotation an image sensor to have pixel rows substantiallyparallel to an axis-defining feature;

FIG. 12 is a schematic representation of an image stitching technique inaccordance with this invention; and

FIG. 13 is a schematic representation of a rotation technique employinga mosaic of images (image data).

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

Contrary to the prior art, the present invention does not seek to aligna desired x′-direction with the x-direction of the XY stage, but insteadpermits a desired x′-direction for scanning across the specimen to beout of alignment with the x-direction of movement of the XY translationstage. The present invention rotates the image sensor to achieve thedesired alignment, thereby creating an offset between the XY directionsof the translation stage and the pixel rows and columns of the imagesensor. The present invention also determines the angle of offset orslope of the desired x′-direction relative to the x-direction of the XYtranslation stage. With the angle of offset/slope known, and the pixelrows aligned with the desired scanning direction, i.e., thex′-direction, the XY translation stage can be controlled to move thespecimen relative to the image sensor to different positions along thedesired x′-direction without a substantial shift of the image sensorrelative to the specimen in a y′-direction, the y′-direction beingorthogonal to the x′ direction of the specimen.

The present invention also provides a method to accurately and preciselyalign the pixel rows of an image sensor with the x-direction of the XYtranslation stage. This alignment then leads to a method for determiningthe angle of offset/slope of a desired x′-direction of a specimenrelative to the x-direction of the XY translation stage.

The general processes of the present invention are disclosed in variousembodiments herein and, once the general physical conditions of theprocess are established, the process can be carried out in an automatedmanner with an appropriately configured microscopy system and relatedcomputer processing and microscopy techniques such as computer vision,motion control and measurement, and the like. As used herein “computervision” is to be understood as covering algorithms for image processing,pattern recognition, computer vision or other known techniques for imageanalysis. First, aspects of the general microscope apparatus aredisclosed, with methods of the present invention being disclosedthereafter.

FIGS. 1A and 1B show an embodiment of a reflected light microscopesystem 2 as an example of the present invention, noting that theinvention is equally applicable to other types of microscopes such astransmitted light microscopes, inverted microscopes, electronmicroscopes and others. Standard parts of the typical microscope system2 include a stand 4, a vertical illuminator 6, a camera 8, a lens tube10, a nosepiece 12, an objective lens 14, a z-axis focus arm 16 and anXY translation stage 18. These components are well known to thosefamiliar with the art. A camera rotator 20 works together with the XYtranslation stage 18 to achieve unique methods of scanning and/orimaging in accordance with this invention.

XY translation stages are well known in the art. They can be driven bystepper, servo, or linear motors, among others. The configuration of anXY translation stage is typically that of affixing one single axis stageto the z-axis focus arm 16 and affixing a second single axis stage tothe first stage with axis of translation being 90 degrees to each other,though minor errors in orthogonal alignment of the X and Y stage areexperienced in practice. Orthogonal alignment is a generally known termand addresses the fact that, for the two stages to travel preciselyalong the x and y axes, the line of travel for the y-axis must beorthogonal to the line of travel of the x-axis. If the two travel linesare not orthogonal, x-axis travel creates a position error in they-direction. An orthogonality error can be expressed as the degrees ofoffset between the theoretical x-axis direction and the direction ofempirical x-axis travel in light of the position error that occurs inthe y-direction. It can also be expressed as a y position offset perx-direction travel length (e.g., 10 micron y position shift per 400 mmx-direction travel).

FIGS. 2A, 2B, and 2C provide an example of a typical configuration of anXY translation stage. FIG. 2A represents the front view of a single axisstage 18 a comprising a drive motor 23, a drive screw 24, and a stand 26with cushion blocks 26 a and 26 b. The cushion blocks 26 a and 26 bcontain bearings and retainers (not shown) to support the drive screw24, which is attached to the drive motor 23 by a coupling (not shown).FIG. 2B is a top down view of the single axis stage 18 a. In thisembodiment, guide rods 28 a and 28 b are present to provide stabilityand guidance to the travel of specimen chuck 30. The specimen chuck 30contains a linear bearing (not shown) through which the guide rodstravel, and a ball nut 32 that propels the chuck 30 along the drivescrew 24 depending upon the direction of rotation of the drive screw 24.The stand 26 is provided with mounting holes 34 a and 34 b to attach tothe z-axis focus arm. A first single axis stage 18 a and a second singleaxis stage 18 b can be joined to create an XY translation stage asgenerally known in the art and shown in FIG. 2C, wherein stage 18 bholds stage 18 a, which holds the specimen chuck 30.

This particular XY translation stage is provided as an example only, andit will be appreciated that other configurations of XY translationsstages existing or hereafter created may be found useful in the presentinvention. As will be appreciated from the disclosure herein, it is onlynecessary that a translation stage used for the present invention becapable of allowing for precise control of the XY stage and providingprecise position information of the X and Y stages. For example, intranslation stages utilizing screw drives, feedback may be in the formof rotary encoders providing a signal that is directly proportional tothe distance traveled. In other translation stages, a linear encoder maybe used to provide direct feedback of the position of the stage.

FIGS. 3A and 3B provide more details of a camera rotator 20, such asthat shown in FIG. 1A. It should be noted that this configuration isshown for example only and that other configurations resulting in therotation of the camera with respect to the XY stage are possible. Here arotator housing 36 is provided with flange 38 so that it may be attachedto lens tube 10 and held in place by locking screws 40 a and 40 b. Arotational camera mount 42 is held in the housing 36 and free to rotatewithin bearings 55. A connector 46 serves to attach the camera 8 to thecamera mount 42. A drive motor 48 together with a pulley 50, a drivebelt 52 and camera rotator pulley 54 provide means to rotate the camera8. A rotary encoder, potentiometer, stepper motor control or othersimilar device can be used so that the angular rotation a is known.These would provide a signal output (for example to processor 22) thatis proportional to the degree of rotation, such that a highly accurateangle of offset is determined from that signal. In some embodiments, arotary encoder is provided within the motor assembly to provide theexact degree of rotation of the camera.

It should be appreciated that all adjustable parts of the microscope canbe controlled by additional appropriate hardware and one or moreprocessors. A processor 22 is shown here as controlling the camerarotator 20, the camera 8, and the XY translation stage 18 (and thez-axis focus arm 16), and appropriate hardware and software isimplicated and generally denoted by processor 22. It will be appreciatedthat multiple processors could be employed, and the same in encompassedby the simple use of processor 22 in the Figures. Other aspects of themicroscope system 2 can also be so controlled. The software isprogrammed to control X, Y and Z movement of the XY translation stage18, as well as rotation of the camera 8 and the activation of the camerato record image data through the image sensor 56. Focusing can beautomated by known methods as well. As is known in the art, the stagetravel can be precisely known and controlled with rotary or linearencoders. The camera rotation can be precisely known by rotary encoders,potentiometers, stepper motor control or other. These precisepositioning devices are used to provide input to software assisting incarrying out the invention. A computer, programmable controller or otherprocessor, such as processor 22, is employed to analyze the inputs andprovide control output to the stage and rotator.

The camera 8 may contain an image sensor 56 (such as a CCD or CMOSsensor) used to capture images (or image data) of a specimen S. As usedherein, an “image” does not require the actual production of an imageviewable by an observer, and it simply requires the taking of digitaldata that can be used to create the desired image. Thus “image” and“image data” are used herein without significant distinction. An imagesensor is comprised of pixels. A typical sensor may have between 0.6 and10 megapixels or more. The size of the pixels vary and are typicallybetween 3 and 10 micrometers (um) and a typical sensor size may bebetween less than 25 square mm to greater than 800 square mm. Arepresentation of the image sensor 56 is shown in FIG. 4. It isimportant to realize that the image sensor's field of view is a functionof the magnification of the microscope objective lens 14. An imagesensor that is 10×12 mm will have a field of view that is 0.10×0.12 mmat 100× magnification. It can be appreciated that the magnified field ofview is much smaller than the specimen to be examined.

As mentioned, the process carried out includes two major steps, a stepof rotating the image sensor such that the pixel rows of the imagesensor are substantially parallel with the desired x′-direction (i.e.,the desired scanning direction) of the specimen, and a step ofdetermining the angle of offset of the desired x′-direction as comparedto the x-direction of the XY translation stage. In different embodimentsherein, sometimes these steps are separate and distinct, and sometimesthey overlap. In accordance with one embodiment, a process for aligningthe pixel rows of the image sensor with the X-direction of the XYtranslation stage is first practiced to provide an accurate referenceposition of the image sensor prior to rotating it to have pixel rowsaligned with the x′-direction of the specimen.

With the pixel rows substantially parallel to the x′-direction, and withthe angle of offset known, advantageous scanning across the specimen canbe achieved by establishing a first position for the specimen relativeto the image sensor, the first position placing at least a portion ofthe specimen within the field of view of the image sensor; and movingthe specimen with the XY translation stage to a second position alongthe desired x′-direction, wherein the second position places at least asecond portion of the specimen within the field of view of the imagesensor, and the second position is not substantially shifted in ay′-direction of the specimen, the y′-direction being orthogonal to thex′-direction of the specimen. In the step of moving, the movement isbased upon the angle of offset determined in said step of determining.

In some embodiments, it is precise to state that an angle of offset isemployed, but it will be appreciated that a slope of the desiredx′-direction as compared to the x′-direction of the XY translation stagecould instead be employed, and, for purposes herein the “angle ofoffset” can be expressed or conceptualized as either a slope (m) or anangle (degrees), the angle of a line of slope m relative to a base linebeing tan−1(m). That is, knowing slope, the angle can be calculated, andvice versa.

A first embodiment of the invention is described with reference to FIGS.5-8, wherein relevant portions of a microscope system 2 are shown andhelp explain the general scanning conditions being addressed by thepresent embodiment. A specimen S is positioned on a specimen chuck 30positioned in proximity of an image sensor 56 of camera 8. The imagesensor 56 is in a fixed position rotatable about its center axis so thatthe XY translation stage 18, represented by the guide rods, and thespecimen S, is moveable to position the specimen S under the imagesensor 56. In some embodiments, the camera 8 and image sensor 56 arepart of a microscope, and it will be appreciated that the image sensor56 can be employed to record image data reaching the image sensor 56through the objective lens 14 and other well-known microscopecomponents. Those of ordinary skill in the art do not require furtherdisclosures beyond the schematics presented here in order to appreciatehow the use of microscopes are implicated.

In this particular example of FIGS. 5-8, it is desired to scan thespecimen S along a desired x′-direction of the specimen. Notably, thedesired x′-direction of the specimen is angularly offset from the x andy movement directions of specimen chuck 30 (as represented by the x andy arrows), and, therefore, the desired x′-direction defines a sloperelative to the x- and y-directions of the stage 18. In this example thedirection of x′ travel is indicated by the line drawn between alignmentmark 60 and alignment mark 62. For purposes of disclosure, the angle ofoffset is shown to be quite significant, and it can be, but it shouldalso be appreciated that the present invention will often be employedwhere the angle of offset is very small, as angles of offset of even atiny fraction of a degree can cause significant problems for scanningand/or stitching images at high magnification.

As seen in FIG. 4, and as generally known in the art, the image sensor56 includes pixel rows and pixel columns, which are schematicallyrepresented as pixel rows P(1,1) to P(j,1) and pixel columns P(1, 1)through P(1, i). As noted in the background, the camera 8 and its imagesensor 56 are typically mounted in an attempt to place the pixel rowsparallel to the x-direction of the translation stage 18/specimen chuck30, thus also placing the pixel columns parallel to the y-direction ofthe translation stage 18 and specimen chuck 30. However due to machiningtolerances and limitations in achieving perfect alignment whetherthrough automated or, more typically, visual methods, the rows andcolumns of the image sensor 56 are usually out of alignment with the x-and y-directions of the translation stage 18. Thus, in this embodiment,the sensor rows and columns are not parallel to either the x′ or y′desired direction of scanning.

In contradistinction to the prior art, the present invention rotates thecamera 8 and thus the image sensor 56 contained within the camera 8 soas to place the pixel rows in a position parallel with the desiredx′-direction of the specimen. In some embodiments, relative rotation canbe accomplished by rotating the image sensor, the camera holding theimage sensor, or the microscope holding the camera or through any otherappropriate manipulation of a component of the system.

In the embodiment of FIG. 5, the angle of offset is determined byassessing a slope between two reference marks, herein alignment marks,on the specimen S. The rotation of the image sensor to place pixel rowssubstantially parallel to the desired x′-direction can be practicedbefore or after assessing the slope, and various methods for suchrotation can be practiced.

In FIG. 5, specimen S is seen to have two alignment marks 60 and 62inscribed. The direction of the line through marks 60 and 62 defines thedesired direction of scanning x′. For reference, orthogonal center linesare shown through image sensor 56, and their intersection marks theimage sensor center C. FIG. 6A illustrates moving specimen chuck 30 to aposition such that alignment mark 60 is positioned in the center ofimage sensor 56. This movement and positioning is determined usingcomputer vision and standard translation stage motion control. Thecoordinates of the XY translation stage 18 are recorded and set as theorigin for future movements. For example, the processor 22 can recordand analyze position and movement data in accordance with the teachingherein. Using computer vision and motion control (e.g., via processor22) the XY stage 18 is repositioned so that alignment mark 62 iscentered in the image sensor 56 as shown in FIG. 6B. The distance tomove the stage in the x-direction and y-direction is determined andrecorded or otherwise retained for further processing in accordance withthe teaching herein. FIG. 7A illustrates this movement and is markedwith the change in the x-direction as ΔX and the change in they-direction as ΔY.

Although the alignment marks 60, 62 are focused onto the center C of theimage sensor 56 in order to assess ΔX and ΔY, it will be appreciatedthat it is possible to designate any pixel or set of pixels as thetarget pixel(s) for placing the alignment marks and assessing ΔX and ΔY.Thus it is sufficient to place the alignment marks 60 so as to overlapwith one or more target pixels of the image sensor 56, and thereaftermove the specimen to place the alignment mark 62 so as to overlap withthe same one or more target pixels; thereafter assessing the x and ymovement to obtain ΔX and ΔY.

In some embodiments, the alignment marks are smaller than a pixel, andare thus targeted on a single pixel to assess ΔX and ΔY. In otherembodiments, the alignment marks encompass multiple pixels. In someembodiments, the alignment marks encompass multiple pixels, and thecenter of the alignment mark is calculated and used for positioning in atarget pixel (such as the center C used in the example). The center canbe calculated through computer vision.

Instead of using alignment marks purposefully placed on the specimen, insome embodiments it is possible to employ component features on thespecimen S such as micro circuitry components (in some embodiments) orphotolithographic features (in other, non-limiting embodiments) thatextend in the desired x′-direction. Identifiable component featureswould be used in the same manner as alignment marks.

Knowing ΔX and ΔY provides the slope (m), which is ΔY/ΔX. With theslope, the desired direction of movement x′ is defined as compared tothe x-direction and y-direction of the XY translation stage. The linedefined by slope ΔY/ΔX and going through alignment marks 60, 62 forms anangle α relative to the line extending in the x-direction and extendingthrough the alignment mark 60. Referring to FIG. 8, the angle α can becalculated as tan−1(m). Knowing the angle α, any movement in thex′-direction from any point of origin (e.g., a point encompassed byalignment mark 60) can be calculated. For example, in FIG. 8, to move adistance of d′ in the x′-direction from the point of origin at alignmentmark 60 the specimen chuck can move: ΔX=d′(cos(α)); ΔY=d′(sin(α)). Theorigin can be set at any location within the plane defined by themaximum travel of the translation stage 18 in the x-direction and they-direction. It can be appreciated that the same procedures can be usedto calculate movement in the y′-direction. It can also be appreciatedthat other mathematical techniques may be used to calculate travel inthe x′- and y′-directions.

Being able to move precisely to different positions along thex′-direction without a substantial shift in the y′-direction allows foraccurate scanning in the x′-directions and facilitates accuratestitching of multiple images or image data recorded by the image sensor,particularly when the rows of pixels of the image sensor aresubstantially parallel to the x′-direction. Thus, in the presentembodiment, either before or after determining the slope/angle of offsetas noted above, the image sensor is rotated to orient the rows of pixelssubstantially parallel to the x′-direction, and some methods for doingso are next disclosed.

In the particular method represented in the drawings, and particularlyFIG. 7B, the sensor is rotated by rotation Ra so as to align the pixelrows with desired scanning direction x′. In this embodiment, therotation occurs after the determination of ΔX and ΔY, but it will beappreciated that, in other embodiments, the image sensor could first bealigned and then ΔX and ΔY can be determined through the movementsdescribed above. Rotation techniques are disclosed herein below.

In some embodiments, as generally represented in FIGS. 11A, 11B, and11C, the rotating of the image sensor includes identifying anaxis-defining feature 66 on the specimen S, the axis-defining feature 66having detectable shape running in a desired x′-direction. Theaxis-defining feature 66 is so named because it is provided to definethe desired scanning direction x′. Computer vision is used to orient thepixel rows parallel to the x′-direction of the specimen based on thedetectable shape of the axis-defining feature 66. A rectangular shape isshown as the axis-defining feature 66.

Another rotation technique is shown in FIG. 13, and includes forming amosaic of overlapping images (m1, m2, m3, m4) encompassing both firstalignment mark 60 and second alignment mark 62, and using computervision to calculate a reference line 70 between the two alignment marks(this line also being the desired x′-direction) and align the pixel rowsof the image sensor with the reference line 70. In FIG. 13, the specimenhas been moved relative to the image sensor 56 to take a plurality ofimages (m1, m2, m3, m4), and these images (i.e., image data) are capableof being processed by a computer to define the reference line 70 betweenthe alignment mark 60 and the alignment mark 62. With this composite ofimage data, computer vision is employed to align the pixel rows of theimage sensor with the reference line 70 as calculated by the computer(e.g., processor 22).

For example, from the position of image m1, taking into account theposition of alignment mark 60, the specimen is moved in the x-directionin incremental distances less than the width of the field of view of theimage sensor. Here the distance is 75% of the width (i.e., moved 3pixels in a total of 4) just for ease of depicting the concept indrawings. However, in some embodiments, these increments (including yincrement movements described below) can be between 5 and 50% of the(width or height dimension of the) field of view. In other embodiments,the increments are between 10% and 30% of the field of view, and inother embodiments, between 10 and 20% of the field of view. The specimenis moved at such increments in the x-direction until it has been moved adistance suitable for aligning the image sensor 56 under the alignmentmark 62. At each incremental movement an image is taken (e.g., m1, m2,m3, m4). The specimen is then moved in the y-direction until alignmentmark 62 is within the field of view of the sensor. An image is taken ateach increment (e.g., m5, m6). Using standard image stitchingtechniques, a composite image is obtained showing alignment marks 60 and62 in the composite image. Again, with this composite of image data,computer vision is employed to align the pixel rows of the image sensorwith the reference line 70.

With the pixel rows aligned (regardless of the method employed to do so)the slope/angle of offset can be employed as noted with respect to FIG.8 to accurately move/scan alone the x′-direction.

With respect to orienting pixel rows parallel to the desiredx′-direction, it will be appreciated that a perfectly parallelrelationship is likely theoretical only, especially when considering thepotential for working at high magnification (where small angular offsetsare more easily appreciated). The present invention seeks to align thepixel rows with the desired x′-direction so that there is extremely lowor no degree of offset between the x′-direction of the specimen and thedirection the pixel rows extend. This is similar to the concerns of“orthogonality error” described above with respect to XY translationstages. In some embodiments, it is sufficient herein that the pixel rowsbe less than 0.002 degrees off of the desired x′-direction. In someembodiments, the pixel rows are less than 0.0015 degrees off of thedesired x′-direction, in other embodiments, less than 0.001 degrees, inother embodiments, less than 0.0005 degrees, in other embodiments, lessthan 0.00025 degrees, in other embodiments, less than 0.0002 degrees, inother embodiments, less than 0.00015 degrees, in other embodiments, lessthan 0.0001 degree, and, in other embodiments, less than 0.00005degrees. In sum, the recitations herein regarding alignment do notrequire absolute perfect alignment, but rather a substantial alignment(or substantially parallel relationship) suitable for the purpose forwhich the present invention is provided. In some embodiments, theinvention serves to substantially reduce orthogonality error evident athigh magnifications, even on the order of 100× or higher. In particular,the present invention provides a highly accurate alignment suitable forscanning along a desired x′-direction without a significant shift in they′-direction, even at high levels of magnification.

A second embodiment of the invention is shown in FIGS. 9-11, and relatesfirst to a method of aligning the image sensor 56 with the translationstage 18 so that the pixel rows are substantially parallel to thex-direction. A first step of this embodiment thus requires the sensor 56be aligned with the movement of the translation stage 18 (i.e., pixelrows and columns parallel to the x-direction travel and the y-directiontravel). FIG. 9A shows the translation stage with the specimen chuck 30at a position left of the image sensor 56. In some embodiments, thespecimen chuck 30 a has a reference mark 64 imprinted thereon providedto be viewed and located by computer vision. In other embodiments, aspecimen S placed on the specimen chuck 30 a has a reference markthereon, such that this method envisions either the specimen chuck or aspecimen placed thereon including a reference mark, but the method isspecifically disclosed with respect to images showing the reference mark64 on the specimen chuck 30 a. When implementing this method with areference mark on a specimen placed on the specimen chuck, the relativepositions between the chuck and the specimen should not be allowed tochange during movement of the specimen chuck.

Per FIG. 9B, the specimen chuck 30 is moved in the x-direction untilreference mark 64 is in the field of view of the image sensor 56. Asnoted previously, each pixel has a unique location designated by P(j,i).In FIG. 9B that mark 64 has been imaged and, for the purpose of example,is located in P(13,4). As seen in FIG. 9C, the specimen chuck 30 ismoved in the x-direction by the XY translation stage 18 so thatreference mark 64 is laterally moved (x-direction) relative to the imagesensor 56 but still in the image sensor field of view. The referencemark 64 is now position in pixel P(9,17). It should be noted that theexample shown is not to actual scale. Image sensors can have 1,000s ofrows of pixels and 1,000s of columns of pixels.

After this first lateral movement and imaging, the goal is to rotate thecamera so that when the image sensor 56 scans across the field of viewof the specimen chuck 30, the reference mark 64 is imaged in the samepixel row or rows as it passes across the sensor, i.e., there is nosubstantial change in relative positions in the y-direction. FIG. 10Ashows the image sensor 56 and the relative locations of the referencemark 64 in the image of FIG. 9B (mark at P(13,4)) and the image of FIG.9C (mark at P(9,17)). Center lines shows the x- and y-direction ofmovement of the specimen chuck 30 (as defined by the translation stage18). With the image sensor 56 oriented as in FIGS. 9A-9C, there is avery large y-direction shift during relative movement of the imagesensor 56 and the specimen chuck 30, and this shift can be reduced andeffectively eliminated by a method involving averaging row numbers ofthe reference mark 64 at the two positions. In this example, thereference mark 64 is in row 13 in the image of FIG. 9B, and in row 9 inthe image of FIG. 9C. A line which represents the average (i.e.,(9+13)÷2=11) is shown across row 11 in FIGS. 10a and 10b . Assuming thatthe specimen chuck 30 is still in the second position as indicated inFIG. 9C, the camera 8 and image sensor 56 is rotated by camera rotator 8by rotation Rb (see FIG. 10A) so the reference mark 64 is in row 11. Asseen in FIG. 10B, this step places the rows of pixels in the imagesensor 56 closer to parallel with the x-direction of the specimen chuck30. These steps of imaging the mark across the image sensor at twolocations and then rotating the camera is repeated until the desiredsubstantial alignment is reached between the pixel rows of the imagesensor and the x-direction of the translation stage. In someembodiments, the steps are repeated until the center of the referencemark 64 (as determined by computer vision) remains in a single row ofpixels when scanned across the entire width of the image sensor 56.Notably, averaging row numbers of the reference mark 64 at the twopositions is only one way to iteratively arrive at a highly precisealignment. It is sufficient in other embodiments, that the rotationrepresented by Rb simply place the reference mark 64 in a row numberbetween the two row numbers for the reference mark upon movement of theimage sensor between the two positions.

In some embodiments, the reference mark 64 is smaller than a pixel, andis thus targeted on a single pixel to perform this alignment process. Inother embodiments, the reference mark 64 encompasses multiple pixels. Insome embodiments, the reference mark 64 encompasses multiple pixels, andthe center of the reference mark 64 is calculated and used forpositioning in a target pixel (such as the center C used in theexample). The center can be calculated through computer vision.

In some embodiments, the reference mark 64 is positioned close to theedge of the field of view, but still within the field of view of theimage sensor, in each placement. This provides a more accurateassessment because it employs a longer x-direction travel by which toassess the y-direction shift. For example, per the figures, thereference mark 64 is positioned so that, in the image shown in FIG. 9B,the mark is positioned to the left of center close to the edge of thefield of view but still within the field of view, while the referencemark 64 is then re-positioned so that, in FIG. 9C, the reference mark 64is positioned to the right of center close to the edge of the field ofview but still within the field of view.

After aligning the image sensor 56 to the x movement of the specimenchuck 30, the image sensor 56 is rotated to align image sensor 56 with adesired x′-direction for scanning across the specimen. In thisembodiment specimen S1 represents a specimen that with axis-definingfeature 66 imprinted on it with distinct axis-defining characteristicswhich define the desired x′- and/or y′-directions for scanning acrossthe specimen S1. FIG. 11A shows the specimen S1 place on specimen chuck30 prior to scanning, and line 72 is provided to visually represent thatthe image sensor 56 is aligned with the x and y travel of the specimenchuck 30. As seen in FIG. 11B, the specimen chuck 30 is moved so thatfeature 66 is within the field of view of the sensor 56. In FIG. 11C theimage sensor 56 is rotate by a measured number of degrees, Rc, which isalso α, so as to align the pixel rows of the image sensor with thex′-direction defined by feature 66. As already noted, the image sensorrotation can be precisely known by rotary encoders, potentiometer,stepper motor control or other, such that the angular rotation a isknown.

Alternatively, after aligning the image sensor 56 to the x movement ofthe specimen chuck 30, alignment marks and a mosaic may be employed toidentify the desired x′-direction and rotate the image sensor (as inFIG. 13).

Notably, because the reference mark 64 is associated with the XYtranslation stage, the aligning of the pixel rows of the image sensorwith the x-direction of the XY translation stage needs only be performedonce, and the aligned position can be recorded for future use. Thus, adifferent specimen can be placed on the chuck with a differentorientation and different axis-defining feature, and the image sensorcan be positioned with pixel rows aligned to the x-direction (per theprocess above) and then rotated to the axis-defining feature to find theangle of offset of desired x′-direction of the new specimen.

Starting with the image sensor pixel rows aligned with the x-directionof the XY stage, the subsequent angular rotation of the camera Rc (toplace the pixel rows in the desired x′-direction) is equivalent to adescribed with respect to FIG. 8, and any movement d′ in a desiredx′-direction can now be determined by: ΔX=d′(cos(α)); ΔY=d′(sin(α)).

Regardless of the methods herein employed to align the pixel rows with adesired x′-direction and to determine the slope/angle of offset of thatx′-direction relative to the x-direction of the XY translation stage,once the pixels are so aligned and the slope/angle is determined, thespecimen S can be moved in such a manner that a left border of a firstimage can be accurately stitched to a right border of a second image,where “left” and “right” are defined in the x′-direction. For example,in FIG. 12, a first imaging position is shown by the positioning ofimage sensor 14 at p1 and a second imaging position is shown by thepositioning of image sensor 14 at p2, wherein the left border of imagingsensor 14 at p1 is aligned with the right border of image sensor 14 atp2, again without any significant shifting in the y′ direction. Itshould also be appreciated that it is acceptable to take images p1 andp2 with an overlapping region, i.e., where columns of pixels at the leftside of the image at p1 overlap with columns of pixels at the right sideof the image at p2. The overlapping portions can assist in the accuratestitching, as known. The present invention, however, by causing analignment of pixel rows with the desired x′-direction, facilitates anaccurate border-to-border stitching of images, which can decrease theentire imaging process by requiring less images and less computation.

It should be appreciated that the various steps herein can, andpreferably are performed automatically by the microscope system 10. Themovement of moveable and rotatable components would be handled byappropriate software and hardware, and computer vision can be employedto identify detectable features such as the alignment marks 60, 62, thereference mark 64, and the axis-defining features 66 that govern theorientation of the image sensor 56 relative to the specimen. Thefocusing and taking of image data can also be automated. This all isrepresented in the figures by processor 22. Thus, the present inventionallows the specimen and XY translation stage to be out of alignment(i.e., desired x′- and y′-directions of the specimen are out ofalignment with the x- and y-directions of the translation stage) anddoes not require manipulation of the specimen to remedy this lack ofalignment. The system self-calibrates, and, knowing the width of animage sensor, can progressively scan the specimen and take discreetimages having aligned borders to then be stitched together to form thecomplete desired image of the specimen.

The general concepts of the present invention are adequately disclosedto those of ordinary skill in the art by the figures and descriptionherein. The detailed disclosure is provided to broadly disclose thosegeneral concepts, but is not necessary for those of ordinary skill inthe art to fully implement the concepts of the present invention. Thisis true even though the drawings are schematic.

Before concluding it should be noted that focusing on the x-direction isthis disclosure is in no way limiting, in that x and y-directions aresimply based on orientations, and the present invention is employed inthe same manner to attend to scanning in a desired y′-direction. Thusreferences to x and y are purely for purposes of having directionalreference.

While particular embodiments of the invention have been disclosed indetail herein, it should be appreciated that the invention is notlimited thereto or thereby inasmuch as variations on the inventionherein will be readily appreciated by those of ordinary skill in theart. The scope of the invention shall be appreciated from the claimsthat follow.

What is claimed is:
 1. A microscopy method for imaging a specimencomprising: rotating an image sensor having pixel rows and pixelcolumns, about its center axis, relative to a specimen on an XYtranslation stage that is movable in an x direction and a y direction,wherein rotating the image comprises: identifying an axis-definingfeature on the specimen running in an x′ direction, wherein the xdirection and the y direction in which the XY translation stage ismovable defines a plane of the XY translation stage, and the x′direction is in the plane of the XY translation stage and angularlyoffset from the x direction of the XY translation stage; and aligningthe pixel rows, using computer vision, substantially parallel to theaxis-defining feature on the specimen.
 2. The method of claim 1, furthercomprising aligning the pixel rows substantially parallel to the xdirection of the XY translation stage, before rotating the image sensor.3. The method of claim 1, wherein the axis-defining feature has adetectable shape running in the x′ direction, and the aligning the pixelrows uses computer vision to align the pixel rows substantially parallelto the detectable shape.
 4. A microscope system comprising: amicroscope; an image sensor, having pixel rows and pixel columns androtatable about its center axis, configured to record image data; an XYtranslation stage that is movable in an x direction and a y direction; aprocessor configured to: rotate an image sensor having pixel rows andpixel columns relative to a specimen; identify an axis-defining featureon a specimen running in an x′ direction wherein the x direction and they direction in which the XY translation stage is movable defines a planeof the XY translation stage, and the x′ direction is in the plane of theXY translation stage and angularly offset from the x direction of the XYtranslation stage; and use computer vision to align the pixel rowssubstantially parallel to the axis-defining feature.
 5. The microscopesystem of claim 4, wherein the processor is further configured to alignthe pixel rows substantially parallel to the x direction of the XYtranslation stage, before rotating the image sensor.
 6. The microscopesystem of claim 4, wherein the axis-defining feature has a detectableshape running in the x′ direction, and the processor is configured touse computer vision to align the pixel rows substantially parallel tothe detectable shape.
 7. A microscopy method for imaging a specimencomprising: rotating an image sensor having pixel rows and pixelcolumns, about its center axis, relative to a specimen on an XYtranslation stage that is movable in an x direction and a y direction,wherein rotating the image comprises: taking a mosaic of images suitablefor calculating a reference line between a first focal feature and asecond focal feature on the specimen and using computer vision to alignthe pixel rows of the image sensor with the reference line.
 8. Amicroscope system comprising: a microscope; an image sensor, havingpixel rows and pixel columns and rotatable about its center axis,configured to record image data; an XY translation stage that is movablein an x direction and a y direction; a processor configured to: rotatean image sensor having pixel rows and pixel columns relative to aspecimen; identify an axis-defining feature on a specimen running in thex′ direction; take a mosaic of images suitable for calculating areference line between a first focal feature and a second focal featureon the specimen; and use computer vision to align the pixel rows of theimage sensor with the reference line.